High Efficiency Video Coding

High Efficiency Video Coding (HEVC), also known as H.265 and MPEG-H Part 2, is a video compression standard designed as part of the MPEG-H project as a successor to the widely used Advanced Video Coding (AVC, H.264, or MPEG-4 Part 10). In comparison to AVC, HEVC offers from 25% to 50% better data compression at the same level of video quality, or substantially improved video quality at the same bit rate. It supports resolutions up to 8192×4320, including 8K UHD, and unlike the primarily 8-bit AVC, HEVC's higher fidelity Main 10 profile has been incorporated into nearly all supporting hardware.

While AVC uses the integer discrete cosine transform (DCT) with 4×4 and 8×8 block sizes, HEVC uses integer DCT and DST transforms with varied block sizes between 4×4 and 32×32. The High Efficiency Image Format (HEIF) is based on HEVC. , HEVC is used by 43% of video developers, and is the second most widely used video coding format after AVC.

Concept

In most ways, HEVC is an extension of the concepts in H.264/MPEG-4 AVC. Both work by comparing different parts of a frame of video to find areas that are redundant, both within a single frame and between consecutive frames. These redundant areas are then replaced with a short description instead of the original pixels. The primary changes for HEVC include the expansion of the pattern comparison and difference-coding areas from 16×16 pixel to sizes up to 64×64, improved variable-block-size segmentation, improved "intra" prediction within the same picture, improved motion vector prediction and motion region merging, improved motion compensation filtering, and an additional filtering step called sample-adaptive offset filtering. Effective use of these improvements requires much more signal processing capability for compressing the video, but has less impact on the amount of computation needed for decompression.

HEVC was standardized by the Joint Collaborative Team on Video Coding (JCT-VC), a collaboration between the ISO/IEC MPEG and ITU-T Study Group 16 VCEG. The ISO/IEC group refers to it as MPEG-H Part 2 and the ITU-T as H.265. The first version of the HEVC standard was ratified in January 2013 and published in June 2013. The second version, with multiview extensions (MV-HEVC), range extensions (RExt), and scalability extensions (SHVC), was completed and approved in 2014 and published in early 2015. Extensions for 3D video (3D-HEVC) were completed in early 2015, and extensions for screen content coding (SCC) were completed in early 2016 and published in early 2017, covering video containing rendered graphics, text, or animation as well as (or instead of) camera-captured video scenes. In October 2017, the standard was recognized by a Primetime Emmy Engineering Award
as having had a material effect on the technology of television.

HEVC contains technologies covered by patents owned by the organizations that participated in the JCT-VC. Implementing a device or software application that uses HEVC may require a license from HEVC patent holders. The ISO/IEC and ITU require companies that belong to their organizations to offer their patents on reasonable and non-discriminatory licensing (RAND) terms. Patent licenses can be obtained directly from each patent holder, or through patent licensing bodies, such as MPEG LA, Access Advance, and Velos Media.

The combined licensing fees currently offered by all of the patent licensing bodies are higher than for AVC. The licensing fees are one of the main reasons HEVC adoption has been low on the web and is why some of the largest tech companies (Amazon, AMD, Apple, ARM, Cisco, Google, Intel, Microsoft, Mozilla, Netflix, Nvidia, and more) have joined the Alliance for Open Media, which finalized royalty-free alternative video coding format AV1 on March 28, 2018.

History

The HEVC format was jointly developed by more than a dozen organisations across the world. The majority of active patent contributions towards the development of the HEVC format came from five organizations: Samsung Electronics (4,249 patents), General Electric (1,127 patents), M&K Holdings (907 patents), NTT ( patents), and JVC Kenwood (628 patents). Other patent holders include Fujitsu, Apple, Canon, Columbia University, KAIST, Kwangwoon University, MIT, Sungkyunkwan University, Funai, Hikvision, KBS, KT and NEC.

Previous work

In 2004, the ITU-T Video Coding Experts Group (VCEG) began a major study of technology advances that could enable creation of a new video compression standard (or substantial compression-oriented enhancements of the H.264/MPEG-4 AVC standard). In October 2004, various techniques for potential enhancement of the H.264/MPEG-4 AVC standard were surveyed. In January 2005, at the next meeting of VCEG, VCEG began designating certain topics as "Key Technical Areas" (KTA) for further investigation. A software codebase called the KTA codebase was established for evaluating such proposals. The KTA software was based on the Joint Model (JM) reference software that was developed by the MPEG & VCEG Joint Video Team for H.264/MPEG-4 AVC. Additional proposed technologies were integrated into the KTA software and tested in experiment evaluations over the next four years.

Two approaches for standardizing enhanced compression technology were considered: either creating a new standard or creating extensions of H.264/MPEG-4 AVC. The project had tentative names ''H.265'' and ''H.NGVC'' (Next-generation Video Coding), and was a major part of the work of VCEG until its evolution into the HEVC joint project with MPEG in 2010.

The preliminary requirements for NGVC were the capability to have a bit rate reduction of 50% at the same subjective image quality compared with the H.264/MPEG-4 AVC High profile and computational complexity ranging from 1/2 to 3 times that of the High profile. NGVC would be able to provide 25% bit rate reduction along with 50% reduction in complexity at the same perceived video quality as the High profile, or to provide greater bit rate reduction with somewhat higher complexity.

The ISO/IEC Moving Picture Experts Group (MPEG) started a similar project in 2007, tentatively named ''High-performance Video Coding''. An agreement of getting a bit rate reduction of 50% had been decided as the goal of the project by July 2007. Early evaluations were performed with modifications of the KTA reference software encoder developed by VCEG. By July 2009, experimental results showed average bit reduction of around 20% compared with AVC High Profile; these results prompted MPEG to initiate its standardization effort in collaboration with VCEG.

Joint Collaborative Team on Video Coding

MPEG and VCEG established a Joint Collaborative Team on Video Coding (JCT-VC) to develop the HEVC standard.

Standardization

A formal joint Call for Proposals on video compression technology was issued in January 2010 by VCEG and MPEG, and proposals were evaluated at the first meeting of the MPEG & VCEG Joint Collaborative Team on Video Coding (JCT-VC), which took place in April 2010. A total of 27 full proposals were submitted. Evaluations showed that some proposals could reach the same visual quality as AVC at only half the bit rate in many of the test cases, at the cost of 2–10× increase in computational complexity, and some proposals achieved good subjective quality and bit rate results with lower computational complexity than the reference AVC High profile encodings. At that meeting, the name ''High Efficiency Video Coding'' (HEVC) was adopted for the joint project. Starting at that meeting, the JCT-VC integrated features of some of the best proposals into a single software codebase and a "Test Model under Consideration", and performed further experiments to evaluate various proposed features. The first working draft specification of HEVC was produced at the third JCT-VC meeting in October 2010. Many changes in the coding tools and configuration of HEVC were made in later JCT-VC meetings.

On January 25, 2013, the ITU announced that HEVC had received first stage approval (consent) in the ITU-T Alternative Approval Process (AAP). On the same day, MPEG announced that HEVC had been promoted to Final Draft International Standard (FDIS) status in the MPEG standardization process.

On April 13, 2013, HEVC/H.265 was approved as an ITU-T standard. The standard was formally published by the ITU-T on June 7, 2013, and by the ISO/IEC on November 25, 2013.

On July 11, 2014, MPEG announced that the 2nd edition of HEVC will contain three recently completed extensions which are the multiview extensions (MV-HEVC), the range extensions (RExt), and the scalability extensions (SHVC).

On October 29, 2014, HEVC/H.265 version 2 was approved as an ITU-T standard. It was then formally published on January 12, 2015.

On April 29, 2015, HEVC/H.265 version 3 was approved as an ITU-T standard.

On June 3, 2016, HEVC/H.265 version 4 was consented in the ITU-T and was not approved during a vote in October 2016.

On December 22, 2016, HEVC/H.265 version 4 was approved as an ITU-T standard.

Patent licensing

On September 29, 2014, MPEG LA announced their HEVC license which covers the essential patents from 23 companies. The first 100,000 "devices" (which includes software implementations) are royalty free, and after that the fee is $0.20 per device up to an annual cap of $25 million. This is significantly more expensive than the fees on AVC, which were $0.10 per device, with the same 100,000 waiver, and an annual cap of $6.5 million. MPEG LA does not charge any fee on the content itself, something they had attempted when initially licensing AVC, but subsequently dropped when content producers refused to pay it. The license has been expanded to include the profiles in version 2 of the HEVC standard.

When the MPEG LA terms were announced, commenters noted that a number of prominent patent holders were not part of the group. Among these were AT&T, Microsoft, Nokia, and Motorola. Speculation at the time was that these companies would form their own licensing pool to compete with or add to the MPEG LA pool. Such a group was formally announced on March 26, 2015, as HEVC Advance. The terms, covering 500 essential patents, were announced on July 22, 2015, with rates that depend on the country of sale, type of device, HEVC profile, HEVC extensions, and HEVC optional features. Unlike the MPEG LA terms, HEVC Advance reintroduced license fees on content encoded with HEVC, through a revenue sharing fee.

The initial HEVC Advance license had a maximum royalty rate of US$2.60 per device for Region 1 countries and a content royalty rate of 0.5% of the revenue generated from HEVC video services. Region 1 countries in the HEVC Advance license include the United States, Canada, European Union, Japan, South Korea, Australia, New Zealand, and others. Region 2 countries are countries not listed in the Region 1 country list. The HEVC Advance license had a maximum royalty rate of US$1.30 per device for Region 2 countries. Unlike MPEG LA, there was no annual cap. On top of this, HEVC Advance also charged a royalty rate of 0.5% of the revenue generated from video services encoding content in HEVC.

When they were announced, there was considerable backlash from industry observers about the "unreasonable and greedy" fees on devices, which were about seven times that of the MPEG LA's fees. Added together, a device would require licenses costing $2.80, twenty-eight times as expensive as AVC, as well as license fees on the content. This led to calls for "content owners oband together and agree not to license from HEVC Advance". Others argued the rates might cause companies to switch to competing standards such as Daala and VP9.

On December 18, 2015, HEVC Advance announced changes in the royalty rates. The changes include a reduction in the maximum royalty rate for Region 1 countries to US$2.03 per device, the creation of annual royalty caps, and a waiving of royalties on content that is free to end users. The annual royalty caps for a company is US$40 million for devices, US$5 million for content, and US$2 million for optional features.

On February 3, 2016, Technicolor SA announced that they had withdrawn from the HEVC Advance patent pool and would be directly licensing their HEVC patents. HEVC Advance previously listed 12 patents from Technicolor. Technicolor announced that they had rejoined on October 22, 2019.

On November 22, 2016, HEVC Advance announced a major initiative, revising their policy to allow software implementations of HEVC to be distributed directly to consumer mobile devices and personal computers royalty free, without requiring a patent license.

On March 31, 2017, Velos Media announced their HEVC license which covers the essential patents from Ericsson, Panasonic, Qualcomm Incorporated, Sharp, and Sony.

the MPEG LA HEVC patent list is 164 pages long.

Patent holders

The following organizations currently hold the most active patents in the HEVC patent pools listed by MPEG LA and HEVC Advance.

Versions

Versions of the HEVC/H.265 standard using the ITU-T approval dates.
* Version 1: (April 13, 2013) First approved version of the HEVC/H.265 standard containing Main, Main10, and Main Still Picture profiles.
* Version 2: (October 29, 2014) Second approved version of the HEVC/H.265 standard which adds 21 range extensions profiles, two scalable extensions profiles, and one multi-view extensions profile.
* Version 3: (April 29, 2015) Third approved version of the HEVC/H.265 standard which adds the 3D Main profile.
* Version 4: (December 22, 2016) Fourth approved version of the HEVC/H.265 standard which adds seven screen content coding extensions profiles, three high throughput extensions profiles, and four scalable extensions profiles.
* Version 5: (February 13, 2018) Fifth approved version of the HEVC/H.265 standard which adds additional SEI messages that include omnidirectional video SEI messages, a Monochrome 10 profile, a Main 10 Still Picture profile, and corrections to various minor defects in the prior content of the Specification.
* Version 6: (June 29, 2019) Sixth approved version of the HEVC/H.265 standard which adds additional SEI messages that include SEI manifest and SEI prefix messages, and corrections to various minor defects in the prior content of the Specification.
* Version 7: (November 29, 2019) Seventh approved version of the HEVC/H.265 standard which adds additional SEI messages for fisheye video information and annotated regions, and also includes corrections to various minor defects in the prior content of the Specification.
* Version 8: As of August 2021 Version 8 is in "Additional Review" status while Version 7 is in force.

Implementations and products

2012

On February 29, 2012, at the 2012 Mobile World Congress, Qualcomm demonstrated a HEVC decoder running on an Android tablet, with a Qualcomm Snapdragon S4 dual-core processor running at 1.5 GHz, showing H.264/MPEG-4 AVC and HEVC versions of the same video content playing side by side. In this demonstration, HEVC reportedly showed almost a 50% bit rate reduction compared with H.264/MPEG-4 AVC.

2013

On February 11, 2013, researchers from MIT demonstrated the world's first published HEVC ASIC decoder at the International Solid-State Circuits Conference (ISSCC) 2013. Their chip was capable of decoding a 3840×2160p at 30 fps video stream in real time, consuming under 0.1 W of power.

On April 3, 2013, Ateme announced the availability of the first open source implementation of a HEVC software player based on the OpenHEVC decoder and GPAC video player which are both licensed under LGPL. The OpenHEVC decoder supports the Main profile of HEVC and can decode 1080p at 30 fps video using a single core CPU. A live transcoder that supports HEVC and used in combination with the GPAC video player was shown at the ATEME booth at the NAB Show in April 2013.

On July 23, 2013, MulticoreWare announced, and made the source code available for the x265 HEVC Encoder Library under the GPL v2 license.

On August 8, 2013, Nippon Telegraph and Telephone announced the release of their HEVC-1000 SDK software encoder which supports the Main 10 profile, resolutions up to 7680×4320, and frame rates up to 120 fps.

On November 14, 2013, DivX developers released information on HEVC decoding performance using an Intel i7 CPU at 3.5 GHz with 4 cores and 8 threads. The DivX 10.1 Beta decoder was capable of 210.9 fps at 720p, 101.5 fps at 1080p, and 29.6 fps at 4K.

On December 18, 2013, ViXS Systems announced shipments of their XCode (not to be confused with Apple's Xcode IDE for MacOS) 6400 SoC which was the first SoC to support the Main 10 profile of HEVC.

2014

On April 5, 2014, at the NAB show, eBrisk Video, Inc. and Altera Corporation demonstrated an FPGA-accelerated HEVC Main10 encoder that encoded 4Kp60/10-bit video in real-time, using a dual-Xeon E5-2697-v2 platform.

On August 13, 2014, Ittiam Systems announced availability of its third generation H.265/HEVC codec with 4:2:2 12-bit support.

On September 5, 2014, the Blu-ray Disc Association announced that the 4K Blu-ray Disc specification would support HEVC-encoded 4K video at 60 fps, the Rec. 2020 color space, high dynamic range ( PQ and HLG), and 10-bit color depth. 4K Blu-ray Discs have a data rate of at least 50 Mbit/s and disc capacity up to 100 GB. 4K Blu-ray Discs and players became available for purchase in 2015 or 2016.

On September 9, 2014, Apple announced the iPhone 6 and iPhone 6 Plus which support HEVC/H.265 for FaceTime over cellular.

On September 18, 2014, Nvidia released the GeForce GTX 980 (GM204) and GTX 970 (GM204), which includes Nvidia NVENC, the world's first HEVC hardware encoder in a discrete graphics card.

On October 31, 2014, Microsoft confirmed that Windows 10 will support HEVC out of the box, according to a statement from Gabriel Aul, the leader of Microsoft Operating Systems Group's Data and Fundamentals Team. Windows 10 Technical Preview Build 9860 added platform level support for HEVC and Matroska.

On November 3, 2014, Android Lollipop was released with out of the box support for HEVC using Ittiam Systems' software.

2015

On January 5, 2015, ViXS Systems announced the XCode 6800 which is the first SoC to support the Main 12 profile of HEVC.

On January 5, 2015, Nvidia officially announced the Tegra X1 SoC with full fixed-function HEVC hardware decoding.

On January 22, 2015, Nvidia released the GeForce GTX 960 (GM206), which includes the world's first full fixed function HEVC Main/Main10 hardware decoder in a discrete graphics card.

On February 23, 2015, Advanced Micro Devices (AMD) announced that their UVD ASIC to be found in the Carrizo APUs would be the first x86 based CPUs to have a HEVC hardware decoder.

On February 27, 2015, VLC media player version 2.2.0 was released with robust support of HEVC playback. The corresponding versions on Android and iOS are also able to play HEVC.

On March 31, 2015, VITEC announced the MGW Ace which was the first 100% hardware-based portable HEVC encoder that provides mobile HEVC encoding.

On August 5, 2015, Intel launched Skylake products with full fixed function Main/8-bit decoding/encoding and hybrid/partial Main10/10-bit decoding.

On September 9, 2015 Apple announced the Apple A9 chip, first used in the iPhone 6S, its first processor with a hardware HEVC decoder supporting Main 8 and 10. This feature would not be unlocked until the release of iOS 11 in 2017.Apple has chosen HEVC as its next-generation video codec
8 June 2017.

2016

On April 11, 2016, full HEVC (H.265) support was announced in the newest MythTV version (0.28).

On August 30, 2016, Intel officially announced 7th generation Core CPUs (Kaby Lake) products with full fixed function HEVC Main10 hardware decoding support.

On September 7, 2016 Apple announced the Apple A10 chip, first used in the iPhone 7, which included a hardware HEVC encoder supporting Main 8 and 10. This feature would not be unlocked until the release of iOS 11 in 2017.

On October 25, 2016, Nvidia released the GeForce GTX 1050Ti (GP107) and GeForce GTX 1050 (GP107), which includes full fixed function HEVC Main10/Main12 hardware decoder.

2017

On June 5, 2017, Apple announced HEVC H.265 support in macOS High Sierra, iOS 11, tvOS, HTTP Live Streaming and Safari.

On June 25, 2017, Microsoft released a free HEVC app extension for Windows 10, enabling some Windows 10 devices with HEVC decoding hardware to play video using the HEVC format inside any app.

On September 19, 2017, Apple released iOS 11 and tvOS 11 with HEVC encoding & decoding support.

On September 25, 2017, Apple released macOS High Sierra with HEVC encoding & decoding support.

On September 28, 2017, GoPro released the Hero6 Black action camera, with 4K60P HEVC video encoding.

On October 17, 2017, Microsoft removed HEVC decoding support from Windows 10 with the Version 1709 Fall Creators Update, making HEVC available instead as a separate, paid download from the Microsoft Store.

On November 2, 2017, Nvidia released the GeForce GTX 1070 Ti (GP104), which includes full fixed function HEVC Main10/Main12 hardware decoder.

2018

On September 20, 2018, Nvidia released the GeForce RTX 2080 (TU104), which includes full fixed function HEVC Main 4:4:4 12 hardware decoder.

2022

On October 25 2022, Chrome released version 107, which starts supporting HEVC hardware decoding for all platforms "out of box", if the hardware is supported.

Browser support

HEVC is implemented in these web browsers:
* Android browser (since version 5 from November 2014)
* Safari (since version 11 from September 2017)
* Edge (since version 77 from July 2017, supported on Windows 10 1709+ when HEVC video extensions is installed)
* Chrome (since version 107 from October 2022, supported on Windows 8+, macOS 11+, Android 5.0+, ChromeOS, and Linux with supported hardware)

In May 2021, an estimated 18% of browsers in use on desktop and notebook systems were able to play HEVC videos in HTML5 webpages, based on data from StatCounter.

Operating system support

Coding efficiency

The design of most video coding standards is primarily aimed at having the highest coding efficiency. Coding efficiency is the ability to encode video at the lowest possible bit rate while maintaining a certain level of video quality. There are two standard ways to measure the coding efficiency of a video coding standard, which are to use an objective metric, such as peak signal-to-noise ratio (PSNR), or to use subjective assessment of video quality. Subjective assessment of video quality is considered to be the most important way to measure a video coding standard since humans perceive video quality subjectively.

HEVC benefits from the use of larger coding tree unit (CTU) sizes. This has been shown in PSNR tests with a HM-8.0 HEVC encoder where it was forced to use progressively smaller CTU sizes. For all test sequences, when compared with a 64×64 CTU size, it was shown that the HEVC bit rate increased by 2.2% when forced to use a 32×32 CTU size, and increased by 11.0% when forced to use a 16×16 CTU size. In the Class A test sequences, where the resolution of the video was 2560×1600, when compared with a 64×64 CTU size, it was shown that the HEVC bit rate increased by 5.7% when forced to use a 32×32 CTU size, and increased by 28.2% when forced to use a 16×16 CTU size. The tests showed that large CTU sizes increase coding efficiency while also reducing decoding time.

The HEVC Main Profile (MP) has been compared in coding efficiency to H.264/MPEG-4 AVC High Profile (HP), MPEG-4 Advanced Simple Profile (ASP), H.263 High Latency Profile (HLP), and H.262/MPEG-2 Main Profile (MP). The video encoding was done for entertainment applications and twelve different bitrates were made for the nine video test sequences with a HM-8.0 HEVC encoder being used. Of the nine video test sequences, five were at HD resolution, while four were at WVGA (800×480) resolution. The bit rate reductions for HEVC were determined based on PSNR with HEVC having a bit rate reduction of 35.4% compared with H.264/MPEG-4 AVC HP, 63.7% compared with MPEG-4 ASP, 65.1% compared with H.263 HLP, and 70.8% compared with H.262/MPEG-2 MP.

HEVC MP has also been compared with H.264/MPEG-4 AVC HP for subjective video quality. The video encoding was done for entertainment applications and four different bitrates were made for nine video test sequences with a HM-5.0 HEVC encoder being used. The subjective assessment was done at an earlier date than the PSNR comparison and so it used an earlier version of the HEVC encoder that had slightly lower performance. The bit rate reductions were determined based on subjective assessment using mean opinion score values. The overall subjective bitrate reduction for HEVC MP compared with H.264/MPEG-4 AVC HP was 49.3%.

École Polytechnique Fédérale de Lausanne (EPFL) did a study to evaluate the subjective video quality of HEVC at resolutions higher than HDTV. The study was done with three videos with resolutions of 3840×1744 at 24 fps, 3840×2048 at 30 fps, and 3840×2160 at 30 fps. The five second video sequences showed people on a street, traffic, and a scene from the open source computer animated movie ''Sintel''. The video sequences were encoded at five different bitrates using the HM-6.1.1 HEVC encoder and the JM-18.3 H.264/MPEG-4 AVC encoder. The subjective bit rate reductions were determined based on subjective assessment using mean opinion score values. The study compared HEVC MP with H.264/MPEG-4 AVC HP and showed that, for HEVC MP, the average bitrate reduction based on PSNR was 44.4%, while the average bitrate reduction based on subjective video quality was 66.5%.

In a HEVC performance comparison released in April 2013, the HEVC MP and Main 10 Profile (M10P) were compared with H.264/MPEG-4 AVC HP and High 10 Profile (H10P) using 3840×2160 video sequences. The video sequences were encoded using the HM-10.0 HEVC encoder and the JM-18.4 H.264/MPEG-4 AVC encoder. The average bit rate reduction based on PSNR was 45% for inter frame video.

In a video encoder comparison released in December 2013, the HM-10.0 HEVC encoder was compared with the x264 encoder (version r2334) and the VP9 encoder (version v1.2.0-3088-ga81bd12). The comparison used the Bjøntegaard-Delta bit-rate (BD-BR) measurement method, in which negative values tell how much lower the bit rate is reduced, and positive values tell how much the bit rate is increased for the same PSNR. In the comparison, the HM-10.0 HEVC encoder had the highest coding efficiency and, on average, to get the same objective quality, the x264 encoder needed to increase the bit rate by 66.4%, while the VP9 encoder needed to increase the bit rate by 79.4%.

In a subjective video performance comparison released in May 2014, the JCT-VC compared the HEVC Main profile to the H.264/MPEG-4 AVC High profile. The comparison used mean opinion score values and was conducted by the BBC
...